Skip to content

fix(export): add time/size guards to dumpDatabaseRoute for large DBs#232

Open
CuboYe wants to merge 1 commit into
outerbase:mainfrom
CuboYe:fix/large-db-dump-timeout
Open

fix(export): add time/size guards to dumpDatabaseRoute for large DBs#232
CuboYe wants to merge 1 commit into
outerbase:mainfrom
CuboYe:fix/large-db-dump-timeout

Conversation

@CuboYe
Copy link
Copy Markdown

@CuboYe CuboYe commented May 19, 2026

/claim #59

Problem

/export/dump loads the entire database into memory before returning a file. For large databases (>1 GB, approaching Durable Object limits), this causes OOM or hangs past the 30s request timeout with no error message or partial result.

Fix

src/export/dump.ts — three layers of defence:

  1. Pre-export size estimate — runs pragma_page_count + schema size query before loading any row data. Fails fast with 413 if estimated dump >100 MB, with a clear message pointing to per-table fallbacks.

  2. Per-table runtime budget (10s) — checked between rows inside the INSERT loop for each table. Catches tables with millions of rows mid-export, before they consume all memory.

  3. Global time budget (20s) — checked between tables. Stops export cleanly after N tables and tells the caller exactly which table caused the overflow and how many remain.

Error messages are actionable: they name the failing table, the row count if hit, and direct users to /export/csv/:tableName / /export/json/:tableName as a workaround.

What didn't change

  • The core dump format and INSERT generation logic are untouched — only guards added
  • Existing tests still pass (the small-DB path is identical behavior)
  • No new dependencies or R2 bindings required for the in-memory path

Testing

  • Verified pragma_page_count query works in the existing test harness
  • Time budget paths tested by injecting Date.now() mock via Vitest
  • Large-DB error path (413) tested with estimatedBytes threshold mock

- Pre-export estimate via pragma_page_count — fail fast with 413 if >100 MB
- Per-table runtime budget (10s) + row count guard (500k) to prevent unbounded INSERT loops
- Global 20s time budget checked between tables
- Clear error messages pointing to per-table /export/csv/:tableName fallback

Fixes outerbase#59
@CuboYe
Copy link
Copy Markdown
Author

CuboYe commented May 20, 2026

Checking in — any questions on the approach? Happy to adjust the time budget thresholds or add R2 streaming if preferred.

@CuboYe
Copy link
Copy Markdown
Author

CuboYe commented May 20, 2026

Friendly bump — PR is ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants